<!DOCTYPE html>
<html  lang="zh-CN" >
    <head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, minimum-scale=1, initial-scale=1, maximum-scale=5, viewport-fit=cover">
    <title>Prometheus-Grafana-通过作业名关联查询FlinkJVM情况 | 学为我用</title>
    <meta name="description" content="问题阐述使用Flink-prometheus插件推送指标时，在jobManager相关指标和JVM相关指标的Tag中，是没有包含作业名的。至于原因，其实也很好理解，因为jobManager&#x2F;TaskManager是执行单元的物理单元，而Job作业或Task任务是逻辑概念，一个JobManager下有可能多个作业在跑，所以无法直接挂钩。 但是这样一来，对我们的监控来说，就比较不方便了。因此想到了通过">
<meta property="og:type" content="article">
<meta property="og:title" content="Prometheus-Grafana-通过作业名关联查询FlinkJVM情况">
<meta property="og:url" content="https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/index.html">
<meta property="og:site_name" content="学为我用">
<meta property="og:description" content="问题阐述使用Flink-prometheus插件推送指标时，在jobManager相关指标和JVM相关指标的Tag中，是没有包含作业名的。至于原因，其实也很好理解，因为jobManager&#x2F;TaskManager是执行单元的物理单元，而Job作业或Task任务是逻辑概念，一个JobManager下有可能多个作业在跑，所以无法直接挂钩。 但是这样一来，对我们的监控来说，就比较不方便了。因此想到了通过">
<meta property="og:locale" content="zh_CN">
<meta property="og:image" content="https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/s_5.png">
<meta property="og:image" content="https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/s_1.png">
<meta property="og:image" content="https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/s_2.png">
<meta property="og:image" content="https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/s_3.png">
<meta property="og:image" content="https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/s_4.png">
<meta property="article:published_time" content="2020-08-04T06:53:15.000Z">
<meta property="article:modified_time" content="2020-09-10T03:25:12.854Z">
<meta property="article:author" content="章瑱彬">
<meta property="article:tag" content="大数据">
<meta name="twitter:card" content="summary">
<meta name="twitter:image" content="https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/s_5.png">

    
    <link rel="icon" href="/zzbomb/images/favicon_128.ico" type="image/x-icon">

    
<link rel="stylesheet" href="/zzbomb/css/common.min.css">



    
    
    
    
        <link href="//cdn.jsdelivr.net/npm/lightgallery.js@1.1.3/dist/css/lightgallery.min.css" rel="stylesheet">
    
    
    
<link rel="stylesheet" href="/zzbomb/css/iconfont.min.css">

    
<meta name="generator" content="Hexo 4.2.1"></head>

    <body>
        <header class="header header-fixture">
    <div class="profile-search-wrap flex sm:block">
        
        
        <div class="profile sm:text-center md:px-1 lg:px-3 sm:pb-4 sm:pt-6">
            <a id="avatar" role="link" href="https://gitee.com/123zhangzhenbin" class="inline-block lg:w-16 lg:h-16 w-8 h-8 m-2" target="_blank" rel="noopener" rel="noreferrer" >
                <img src="/zzbomb/images/g.jpg" class="rounded-full" alt="avatar">
            </a>
            <h2 id="name" class="hidden lg:block">ZZB</h2>
            <h3 id="title" class="hidden xl:block">Java，大数据，人工智能，区块链</h3>
            
            <small id="location" class="hidden lg:block">
                <i class="iconfont icon-map-icon"></i>
                DongGuan, China
            </small>
            
        </div>
        
        
<div class="search flex-1 flex lg:inline-block sm:hidden lg:px-4 lg:mt-2 lg:mb-4 lg:w-full">
    <form id="search-form" class="my-auto flex-1 lg:border lg:border-solid lg:border-gray-200">
        <div class="input-group table bg-gray-100 lg:bg-white w-full">
            <input id="search-input" type="text" placeholder="搜索" class="inline-block w-full bg-gray-100 lg:bg-white">
            <span class="table-cell">
                <button name="search tigger button" disabled>
                    <i class="iconfont icon-search m-2"></i>
                </button>
            </span>
        </div>
    </form>
        
<script id="search-teamplate" type="text/html" data-path="/zzbomb/content.json">
    <div>
        <div class="search-header bg-gray-400">
            <input id="actual-search-input" model="keyword" ref="input" class="inline-block w-full h-10 px-2 py-1" placeholder="搜索" type="text">
        </div>
        <div class="search-result bg-gray-200">
            {{#each searchPosts}}
            <a href="/{{ path }}" class="result-item block px-2 pb-3 mb-1 pt-1 hover:bg-indigo-100">
                <i class="iconfont icon-file"></i>
                <h1 class="result-title inline font-medium text-lg">{{ title }}</h1>
                <p class="result-content text-gray-600 text-sm">{{{ text }}}</p>
            </a>
            {{/each}}
        </div>
    </div>
</script>

</div>


        <button name="menu toogle button" id="menu-toggle-btn" class="block sm:hidden p-3" role="button" aria-expanded="false">
            <i class="iconfont icon-hamburger"></i>
        </button>
    </div>
    <nav id="menu-nav" class="hidden sm:flex flex-col">
        
        
            <div class="menu-item menu-home" role="menuitem">
                <a href="/zzbomb/.">
                    <i class="iconfont icon-home" aria-hidden="true"></i>
                    <span class="menu-title">首页</span>
                </a>
            </div>
        
        
            <div class="menu-item menu-archives" role="menuitem">
                <a href="/zzbomb/archives">
                    <i class="iconfont icon-archive" aria-hidden="true"></i>
                    <span class="menu-title">归档</span>
                </a>
            </div>
        
        
            <div class="menu-item menu-categories" role="menuitem">
                <a href="/zzbomb/categories">
                    <i class="iconfont icon-folder" aria-hidden="true"></i>
                    <span class="menu-title">分类</span>
                </a>
            </div>
        
        
            <div class="menu-item menu-tags" role="menuitem">
                <a href="/zzbomb/tags">
                    <i class="iconfont icon-tag" aria-hidden="true"></i>
                    <span class="menu-title">标签</span>
                </a>
            </div>
        
        
            <div class="menu-item menu-repository" role="menuitem">
                <a href="/zzbomb/repository">
                    <i class="iconfont icon-project" aria-hidden="true"></i>
                    <span class="menu-title">项目</span>
                </a>
            </div>
        
        
            <div class="menu-item menu-links" role="menuitem">
                <a href="/zzbomb/links">
                    <i class="iconfont icon-friend" aria-hidden="true"></i>
                    <span class="menu-title">友链</span>
                </a>
            </div>
        
        
            <div class="menu-item menu-about" role="menuitem">
                <a href="/zzbomb/about">
                    <i class="iconfont icon-cup" aria-hidden="true"></i>
                    <span class="menu-title">关于</span>
                </a>
            </div>
        
        
<div class="social-links flex sm:flex-col lg:hidden mt-5">
    
        <span class="social-item text-center">
            <a href="https://gitee.com/123zhangzhenbin" target="_blank" rel="noopener">
                <i class="iconfont social-icon icon-github"></i>
                <span class="menu-title hidden lg:inline">menu.github</span>
            </a>
        </span>
    
</div>


    </nav>
</header>

        <section class="main-section">
            
    <main class="flex-1 px-4 py-12 md:px-5 lg:px-8 lg:py-4 relative min-h-screen">
    

    <article class="content article article-archives article-type-list" itemscope="">
        <header class="article-header">
            
    
        <h1 class="article-title text-lg" itemprop="name">
            Prometheus-Grafana-通过作业名关联查询FlinkJVM情况
        </h1>
    



            <p class="article-meta mb-3 text-xs">
                <span class="article-date">
    <i class="iconfont icon-calendar-check"></i>
	<a href="/zzbomb/2020/08/04/202008041452-promethues-join-query/" class="article-date">
	  <time datetime="2020-08-04T06:53:15.000Z" itemprop="datePublished">8月 4</time>
	</a>
</span>

                

                
    <span class="article-tags">
    <i class="iconfont icon-tag"></i>
    <a class="article-tag-link" href="/zzbomb/tags/%E5%A4%A7%E6%95%B0%E6%8D%AE/" rel="tag">大数据</a>
  </span>


                <span class="_partial/post-comment"><i class="icon icon-comment"></i>
                    <a href="/zzbomb/2020/08/04/202008041452-promethues-join-query/#comments" class="article-comment-link">
                        评论
                    </a>
                </span>
                

            </p>
        </header>
        <div class="marked-body article-body">
            <h2 id="问题阐述"><a href="#问题阐述" class="headerlink" title="问题阐述"></a>问题阐述</h2><p>使用Flink-prometheus插件推送指标时，在jobManager相关指标和JVM相关指标的Tag中，是没有包含作业名的。至于原因，其实也很好理解，因为jobManager/TaskManager是执行单元的物理单元，而Job作业或Task任务是逻辑概念，一个JobManager下有可能多个作业在跑，所以无法直接挂钩。</p>
<p>但是这样一来，对我们的监控来说，就比较不方便了。因此想到了通过即有job又有job_name的指标来进行关联。具体方法是通过label_replace方法来填充查询tag</p>
<h2 id="label-replace"><a href="#label-replace" class="headerlink" title="label_replace"></a>label_replace</h2><p>语法如下</p>
<figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs stylus"><span class="hljs-function"><span class="hljs-title">label_replace</span><span class="hljs-params">(query语句, <span class="hljs-string">"new_label"</span>, <span class="hljs-string">"$1new_string$2"</span>, <span class="hljs-string">"old_label"</span>, <span class="hljs-string">"(.*)string(.*)"</span>)</span></span><br></code></pre></td></tr></table></figure>
<p>这个语句的意思是，对old_label进行二次处理，假设old_label的语法是(.<em>)string(.</em>)，前一个括号里匹配的是$1，后一个括号里匹配的是$2，它们通过string分割。再将$1和$2使用“$1new_string$2”的语法进行重新拼接，最终形成了一个new_label。</p>
<h2 id="通过关联查询找到JobManager的堆内存"><a href="#通过关联查询找到JobManager的堆内存" class="headerlink" title="通过关联查询找到JobManager的堆内存"></a>通过关联查询找到JobManager的堆内存</h2><p>比如说，现在有这样两个指标数据</p>
<figure class="highlight routeros"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><code class="hljs routeros">flink_jobmanager_Status_JVM_Memory_Heap_Committed&#123;<span class="hljs-attribute">exported_job</span>=<span class="hljs-string">"flink-joba0b2a12535861f4dadcb9024b6a2e7ad"</span>, <span class="hljs-attribute">host</span>=<span class="hljs-string">"hbase01_data_lan"</span>, <span class="hljs-attribute">instance</span>=<span class="hljs-string">"pushgateway"</span>, <span class="hljs-attribute">job</span>=<span class="hljs-string">"pushgateway"</span>&#125;<br><br>flink_jobmanager_job_uptime&#123;<span class="hljs-attribute">exported_job</span>=<span class="hljs-string">"flink-joba0b2a12535861f4dadcb9024b6a2e7ad"</span>, <span class="hljs-attribute">host</span>=<span class="hljs-string">"hbase01_data_lan"</span>, <span class="hljs-attribute">instance</span>=<span class="hljs-string">"pushgateway"</span>, <span class="hljs-attribute">job</span>=<span class="hljs-string">"pushgateway"</span>, <span class="hljs-attribute">job_id</span>=<span class="hljs-string">"4efcf8fcda8c988736371116028d5dad"</span>, <span class="hljs-attribute">job_name</span>=<span class="hljs-string">"serilog_takeout_to_hbase"</span>&#125;<br></code></pre></td></tr></table></figure>
<p>第一条指标是jobmanager的JVM堆内存的committed使用量<br>第二条指标是jobmanager下各任务的启动时间</p>
<p>可以看到flink_jobmanager_Status_JVM_Memory_Heap_Committed只有exported_job，但是通过exported_job很难直接定位是那个jobmanager<br>而第二条flink_jobmanager_job_uptime不仅有exported_job还有job_name，所以想办法通过flink_jobmanager_job_uptime指标去关联查询，就可以通过job_name取flink_jobmanager_Status_JVM_Memory_Heap_Committed的指标值了。</p>
<p>下面这个表达式，就是通过job_name获取对应jobmanager的JVM_Heap_Committed指标指</p>
<figure class="highlight stylus"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><code class="hljs stylus"><span class="hljs-function"><span class="hljs-title">label_replace</span><span class="hljs-params">(flink_jobmanager_Status_JVM_Memory_Heap_Committed, <span class="hljs-string">"exported_job"</span>, <span class="hljs-string">"$1"</span>, <span class="hljs-string">"flink_jobmanager_job_uptime&#123;job_name='$&#123;job_name&#125;'&#125;"</span>, <span class="hljs-string">"(.*)"</span>)</span></span><br></code></pre></td></tr></table></figure>
<p>效果图如下<br><img src="/zzbomb/2020/08/04/202008041452-promethues-join-query/s_5.png" alt="5"></p>
<h2 id="Grafana下Flink作业监控的配置"><a href="#Grafana下Flink作业监控的配置" class="headerlink" title="Grafana下Flink作业监控的配置"></a>Grafana下Flink作业监控的配置</h2><p>导入以下配置JSON<br><a href="/zzbomb/download/202008041452-prometheus-join-query/grafana.json">grafana.json</a></p>
<p>最终效果图如下<br><img src="/zzbomb/2020/08/04/202008041452-promethues-join-query/s_1.png" alt="1"><br><img src="/zzbomb/2020/08/04/202008041452-promethues-join-query/s_2.png" alt="2"><br><img src="/zzbomb/2020/08/04/202008041452-promethues-join-query/s_3.png" alt="3"><br><img src="/zzbomb/2020/08/04/202008041452-promethues-join-query/s_4.png" alt="4"></p>

        </div>
        
<blockquote class="copyright">
    <p><strong>本文链接 : </strong><a class="permalink" href="https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/">https://123zhangzhenbin.gitee.io/zzbomb/2020/08/04/202008041452-promethues-join-query/</a></p>
    <p><strong>This article is available under <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" target="_blank" rel="noopener noreferrer">Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)</a> License</strong></p>
</blockquote>


    </article>
    
    <section id="comments">
        
    </section>


    

</main>


<aside style="" id="sidebar" class="aside aside-fixture">
    <div class="toc-sidebar">
        <nav id="toc" class="article-toc">
            <h3 class="toc-title">文章目录</h3>
            <ol class="toc"><li class="toc-item toc-level-2"><a class="toc-link" href="#问题阐述"><span class="toc-number">1.</span> <span class="toc-text">问题阐述</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#label-replace"><span class="toc-number">2.</span> <span class="toc-text">label_replace</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#通过关联查询找到JobManager的堆内存"><span class="toc-number">3.</span> <span class="toc-text">通过关联查询找到JobManager的堆内存</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#Grafana下Flink作业监控的配置"><span class="toc-number">4.</span> <span class="toc-text">Grafana下Flink作业监控的配置</span></a></li></ol>
        </nav>
    </div>
</aside>





        </section>
        <footer class="hidden lg:block fixed bottom-0 left-0 sm:w-1/12 lg:w-1/6 bg-gray-100 z-40">
    
    <div class="footer-social-links">
        
            <a href="https://gitee.com/123zhangzhenbin" target="_blank" rel="noopener">
                <i class="iconfont icon-github"></i>
            </a>
        
    </div>
    
    <p style="display: block;text-align: center;font-size: 13px;"><a href="https://beian.miit.gov.cn/" target="_blank">粤ICP备16013026号</a></p>
    
        <p class="footer-custom"><a href="https://beian.miit.gov.cn/" target="_blank">粤ICP备16013026号</a></p>
        <p class="theme-brand">Theme by <a href="https://github.com/fengkx/hexo-theme-purer" target="_blank" rel="nofollow noopener noreferrer noopener">hexo-theme-purer</a></p>

    
</footer>

        <div id="mask" class="hidden mask fixed inset-0 bg-gray-900 opacity-75 z-40"></div>
        <div id="search-view-container" class="hidden shadow-xl"></div>
        
<script src="/zzbomb/js/dom-event.min.js"></script>

<script src="//cdn.jsdelivr.net/npm/yox@1.0.0-alpha.121/dist/standard/prod/yox.min.js"></script>


<script src="/zzbomb/js/search.min.js"></script>



    <script src="//cdn.jsdelivr.net/npm/lightgallery.js@1.1.3/dist/js/lightgallery.min.js"></script>
    
<script src="/zzbomb/js/light-gallery.min.js"></script>





    </body>
</html>
